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INSTITUTE  FOR  DEFENSE  ANALYSES 


PREFACE 


The  prototyping  research  presented  here  is  part  of  a  larger  IDA  study,  Effective  Initiatives  in  Acquiring  Major  Systems. 
The  study  was  co^^cemed  with  the  highly-publicized  cost  and  schedule  overruns  that  have  plagued  defense  programs.  Since  the 
1960s,  the  Defense  E)epartment  and  defense  contractors  have  pioneered  reviews  and  management  initiatives  to  improve  program 
outcomes.  The  study  undertook  a  major  review  of  program  outcomes  in  order  to  determine; 

•  whether  there  is  a  trend  toward  better  outcomes  overall. 
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Broad  guidelines  are  suggested  for  prototyping 


The  lessons  learned  from  prototyping  in  the  acquisition  of  major  defense  systems  are  overwhelmingly  positive. 
Prototyping  helps  developers  and  users  to  understand  the  technical  risks  and  uncertainty  of  the  requirements.  However,  it  is  not 
to  be  suggested  that  prototyping  a  system  is  a  panacea  and  that  subsequent  development  will  be  trouble-free.  Sometimes  what  is 
learned  from  prototyping  is  what  not  to  do.  If  you  then  proceed  into  engineering  and  manufacturing  development  (HMD)  with  a 
new  and  untested  approach,  you  can  still  run  into  great  difficulty. 


-  transition  to  production 


Consideration  of  prototyping  is  especially  timely  now  for  a  number  of  reasons: 

.  A  decreasing  real  defense  budget  increases  pressure  on  weapon  system  developers  to  make  their  programs  more  predictable 
and  financially  viable. 

.  As  a  consequence  of  lower  overall  budgets,  reduced  funding  is  available  for  major  acquisition  programs. 
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The  chart  indicates  that  when  the  DoD  budget  is  high,  relatively  little  prototyping  is  done.  There  was  considerable 
prototyping  in  the  period  of  build-down  following  the  Second  World  War  and  following  the  Kotean  War. 

During  the  early  1960s,  there  was  little  prototyping,  as  the  Kennedy  administration  believed  that  systems  analyses  could 
take  the  place  of  prototypes.  Less  than  a  third  of  major  systems  were  prototyped.  There  was  concern  about  paying  for  a 
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We  examined  four  prototyping  issues. 

.  What  is  a  prototype?  For  purposes  of  this  study,  a  prototype  is  hardware  used  for  testing,  that  is  built  before  engineering  and 
manufacturing  development  (EMD),  or  full-scale  development  (FSD)  in  the  case  of  historical  programs.  A  prototype  may 
have  full  or  partial  capabilities  and  may  be  full  or  partial  scale.  The  definition  and  purposes  of  prototyping  are  addressed  more 
fully  in  the  next  slide. 
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which  the  prototypes  should  encompass  the  entire  weapon  system. 


The  primary  purpose  of  prototyping  is  to  reduce  technical  risk.  Prototypes  can  be  used  to  answer  three  technical 
questions.  The  three  questions,  which  are  not  mutually  exclusive,  are: 

•  Is  the  concept  feasible? — Proof  of  Concept 

•  Does  the  design  work  the  way  it  is  supposed  to  work? — Proof  of  Design 
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We  examined  prototypes  for  major  systems,  corresponding  to  budget  categories  6.2,  6.3A,  and  6.3B.  The  analysis 
focused  on  6.3B  prototypes.  Our  definition  does  not  include  6.4  EMD  test  articles. 


higher  cost  program 


The  program  that  eventually  resulted  in  the  AV-8B  Harrier  aircraft  benefited  from  all  three  categories  of  prototyping.  The 
British  P-1127  program  in  1960  demonstrated  the  concept  and  the  technology  needed  and  was  low-cost. 

Proof  of  design  was  tested  by  the  XV-6A  in  1965,  a  program  that  led  to  the  Marine  Corps  AV-8A  in  1971.  This  was  also 
a  relatively  low-cost  program. 


u  u 


Prototyping  helps  primarily  to  reduce  technical  risks.  However,  sucii  an  impact  is  difficult  to  measure  because  the 
technical  characteristics  are  unique  to  each  equipment  type  and  program.  If  a  program  proceeds  well  from  a  technical  standpoint, 
then  it  is  much  less  likely  to  encounter  schedule  and  cost  problems.  Cost  and  schedule  problems  are  measurable.  The  most 
successful  acquisition  programs  seem  to  have  in  common  several  elements: 

•  Austere  early  research  and  development  (R&D).  It  is  often  possible  to  demonstrate  the  technology  using  relatively 
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Acquisition  managers  get  two  types  of  information  from  prototyping— qualitative  and  quantitative. 

The  qualitative  information  can  include  both  design  and  program  information.  For  example,  in  the  prototype  phase  for  the 
Lightweight  Fighter  (later  the  F-16),  the  fly-by -wire  control  and  autostabilization  system  was  refined  and  proven  to  work  [3]. 
(^estions  such  as:  Can  a  missile  achieve  lock-on?  Can  a  VTOL  aircraft  hover  in  controlled  flight?  What  is  the  ground  effect  of 
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Engineering  changes  (CAD/CAM)— fewer  changes  once  process  was  set  up 


Here  is  one  recent  example  of  the  benefits  of  prototyping  in  an  actual  program:  the  winning  model  for  the  Advanced 
Tactical  Fighter,  the  Lxjckheed  YF-22.  The  general  manager  for  the  program  said  that  the  prototype  phase  yielded  information  that 
probably  prevented  considerable  cost  and  schedule  growth. 
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Development 

-  cost  and  cost  growth 

-  schedule  and  schedule  growth 
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Cost  Growth  was  adjusted  for  quantity 
(Development  Estimate)  and  infiation 


Total  development  cost  growth  significantly  less  for  prototypes 

-  1 7%  vs.  62%  overall 
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particularly  successful  in  terms  of  fewer  unplanned  FSD  articles. 


One  can  see  the  benefits  of  competitive  prototyping  more  clearly  by  looking  at  individual  cases.  In  the  case  of  the  AH-64, 
for  example,  prototyping  aided  in  source  selection  and  allowed  many  changes  in  design  early  in  the  process  [3].  Time  and  again, 
program  management  staffs  have  conunented  that  contractors  are  much  more  responsive  to  requirements  in  the  competitive  phase 
of  a  program  [1]. 

Competition  may  be  more  valuable  in  a  situation  where  there  is  a  great  deal  of  technical  risk,  because  it  allows  contractors 


Early  realism  regarding  achievabie  requirements  is  a  major 
benefit  of  prototyping:  knowing  what  you  can  buy 

-  ATF;  Over  20  changes  in  major  requirements  before 
prototype  fiew 
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should  influence  thinking  about  prototyping. 
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great  speed  at  low  altitude  for  a  relatively  heavy  aircraft.  The  requirement  was  held  to  and  eventually  met,  but  at  a  75  percent 
higher  cost  than  was  originally  planned. 


Some  benefits  of  prototyping  spill  over  into  other  programs,  and  we  could  not  measure  these  in  the  quantitative  analysis. 
They  are  nonetheless  important.  For  example,  the  prototype  competition  in  the  F- 16  program  actually  yielded  two  aircraft.  The 
winning  YF-16  prototype  built  by  General  Dynamics  became  the  F-16,  and  the  losing  YF-17  prototype  built  by  Northrop 
eventually  became  the  F/A-18.  Both  aircraft  first  flew  in  the  late  1970s  and  are  still  in  use  today.  In  addition,  as  discussed  earlier. 
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Requirements  uncertainty:  operational  suitability 
Development  cost  and  schedule  uncertainty:  concept  and  design 
Production  cost,  producibiiity:  add  mission  demonstration 
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Our  quantitative  analysis  was  not  extensive  enough  to  support  development  of  a  cost/benefit  model  for  prototyping. 
Nevertheless,  we  believe  that  we  have  taken  some  impwtant  first  steps  towaid  such  a  model. 

Prototyping  is  a  leveraged  investment.  Spend  small  amounts  of  money  now  to  avoid  large  surprises  later.  As  rules  of 
thumb  for  when  prototyping  makes  sense  relative  to  its  likely  payoff,  we  suggest  that  the  prototype  cost  should  be  less  than  25 
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Prototyping  helps  to  uncover  problems  at  a  time  when  fixing  them  is  relatively  cheap.  In  1972,  David  Packard  said;  “A 
few  months  ago  at  a  meeting  of  military  project  managers,  someone  objected  to  extensive  testing  because  it  would  delay  the 
program.  He  complained  that  testing  showed  up  things  that  needed  to  be  fixed  and  it  took  time  to  fix  them,  and  this  would  delay 
the  initial  operating  capability.  Unless  we  get  rid  of  that  kind  of  thinking  there  will  be  no  hopes.” 
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PROGRAMS  IN  PROTOTYPING  DATABASES 
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